JSSE: fix SSLEngine.unwrap() data delivery and close_notify handling after closeOutbound()#379
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adjusts WolfSSLEngine.unwrap() behavior after closeOutbound() so that, when inbound is still open and the handshake is complete, unwrap continues through the normal receive path to deliver any pending peer application data and process the peer’s close_notify (preventing hangs in close loops). It also adds a regression test that reproduces the SunJSSE interop close sequence that previously could spin indefinitely.
Changes:
- Update
WolfSSLEngine.unwrap()to only route intoClosingConnection()when inbound is already closed or the handshake is not finished, avoiding discarding buffered application data aftercloseOutbound(). - Add
testCloseOutboundWithPendingAppDatato validate app-data delivery during close and ensure both engines fully close within a bounded wrap/unwrap loop.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
src/java/com/wolfssl/provider/jsse/WolfSSLEngine.java |
Refines unwrap shutdown routing so inbound app-data and peer close_notify are still processed after closeOutbound() when appropriate. |
src/test/com/wolfssl/provider/jsse/test/WolfSSLEngineTest.java |
Adds a regression test covering the close sequence with pending app data to prevent hangs and silent data loss. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
rlm2002
left a comment
rlm2002
left a comment
There was a problem hiding this comment.
Claude.Skoll: This new fall-through routes post-closeOutbound() unwrap() into the receive path, which can set BUFFER_OVERFLOW (line ~1652) / BUFFER_UNDERFLOW. But line ~1718 then unconditionally overwrites status = CLOSED whenever outBoundOpen == false. With a nonempty record and an undersized output buffer, that masks the BUFFER_OVERFLOW → the app sees CLOSED/produced=0, never enlarges its buffer, and the stashed app data isn't delivered (verified vs SunJSSE, which returns BUFFER_OVERFLOW).
Consider not overwriting when a BUFFER_OVERFLOW/BUFFER_UNDERFLOW is already set matching the wrap side, which returns overflow before the close-status block.
…se_notify after closeOutbound()
6616699 to
165bd1b
Compare
|
@rlm2002, thanks for the skoll comment, should be fixed now. |
|
@rlm2002, assigning back to you for review. The failing SunJSSE test |
This PR fixes a hang in the SunJSSE interop test
SSLEngine/NoAuthClientAuth.javawhen wolfJSSE is run against wolfSSL master containing 07d4174 (wolfSSL/wolfssl#10863), along with silent discarding of application data received aftercloseOutbound().Problem
When an application calls
SSLEngine.closeOutbound()while inbound is still open,unwrap()routed all processing throughClosingConnection()/wolfSSL_shutdown(). This caused two issues:closeOutbound()were decrypted into native wolfSSL's internal buffer but never delivered to the application (silently discarded).wolfSSL_shutdown()returnsWOLFSSL_SHUTDOWN_NOT_DONEwithout processing incoming records while decrypted application data is still buffered. With unread data pending, the peer's close_notify was never processed,isInboundDone()never returned true, and applications looping on engine close status spun forever.The
NoAuthClientAuth.javaclose sequence (servercloseOutbound()while the client still sends application data followed by close_notify) hits exactly this state.Fix
Only route
unwrap()intoClosingConnection()when inbound is already closed or the handshake is not finished. Otherwise fall through to the normal receive path, which delivers pending application data and processes the peer's close_notify viaRecvAppData().The
WolfSSLSocketpath is unaffected: it makes a singlewolfSSL_shutdown()call per connection (unidirectional close), never reaching the bidirectional branch where the native pending-data check lives.Testing
Added
testCloseOutboundWithPendingAppDatatoWolfSSLEngineTest: drives two engines through the NoAuthClientAuth close sequence with a bounded wrap/unwrap loop, asserting app data sent during close is delivered and both engines fully close.